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Summary 


Life support programs and policies are quite complex and varied. Does the program or 
policy work? Is it worth the cost? Can and should it be implemented elsewhere? It is the 
reputed purpose of evaluation to provide answers to these and related questions. 
Unfortunately, program/policy evaluation has not lived up to expectations. The field of 
evaluation is littered with efforts that do not adequately address the important issues or 
objectives, that do not.employ valid controls for comparison purposes, that rely on 
inadequate measures or include expensive collections of data on measures that are in 
fact never used in the evaluation, that rely on inappropriate measurement methods, or 
that employ inadequate analytic techniques. 


Most, if not all, of the above-cited problems could be mitigated by developing—at the 
beginning of an evaluation effort—a valid and comprehensive evaluation design. 
Although there is no stock evaluation design that can be taken off the shelf and 
implemented without revision, there should be an approach or process by which such 
designs can be developed. Indeed, the author has identified a systems approach—that is 
at once purposeful, systematic, and consistent—for developing valid and comprehensive 
evaluation designs—it has since been successfully employed in a number of evaluation 
efforts. The approach is discussed in this paper, together with its impact on evaluating 
life support programs/policies. Additionally, it is shown that the approach is also 
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applicable to any analysis that seeks to be purposeful, systematic, and consistent. In this 
regard, the words “analysis” and “evaluation” are quite interchangeable. 


1. Introduction 


The 1992 Earth Summit in Rio de Janeiro and the 1997 Kyoto Protocol underscore the 
importance of sustainable development. In order to achieve the sustainable development 
of life support systems, appropriate programs and policies must be identified to ensure 
the security of the world’s water, energy, environment, and food resources, which 
together sustain life on planet Earth. Programs and policies are determined to be 
“appropriate” if their efficacy is ascertained through a purposeful, systematic, and 
consistent evaluation analysis. 


Life support programs and policies are quite complex and varied; they can be behavioral, 
social, cultural, organizational, economic, political, legal, or technological inmature. As 
illustrated in Figure 1, it is convenient to group such programs and policies into two 
categories: those that seek to decrease or mitigate the demand for water, energy, 
environment, and food; and those that seek to increase or renew the supply of these 
same resources. Further, within each resource group, the decrease in demand could be 
effected by consumption, conservation, system management, and public information 
and education, while the increase in supply could be effected by new sources (or 
rejuvenation), pollution control, and new technologies. 


PROGRAMS/ 


LIFE SUPPORT SYSTEMS 
POLICIES 
THAT: WATER ENERGY ENVIRONMENT FOOD 
¢ Consumption (People, © Consumption (People, © Consumption (People, Industries, | © Consumption (People, Animals, 
Animals, Industries, Industries, Transportation, Transportation, Urbanization, Agriculture, Aquaculture) 
Urbanization, Rural Urbanization, Rural Rural Development, Agriculture, 
Development, Agriculture, Development, Agriculture, Aquaculture, Sanitation, 
Aquaculture, Sanitation, Aquaculture, Sanitation, Recreation, Ecosystem) 
Recreation, Ecosystem) Recreation) 
¢ Conservation (Residential, « Conservation (Residential, ¢ Conservation (Atmosphere, ¢ Conservation (Diet, Plant 
Commercial, Industrial, Utility, Commercial, Industrial, Utility, Hydrosphere, Cryosphere, Habitats, Plant Crops, 
* DECREASE Transportation, Agriculture, Transportation, Agriculture, Lithosphere, Pedosphere) Fruits/Nuts, Forests, Animals, 
DEMAND Aquaculture, Urban, Suburban) Aquaculture, Urban, Suburban) Fisheries) 
¢ Systems Management (System | ¢ Systems Management (System « System Management (System * System Management (System 
Modeling/Simulation/Evaluatio Modeling/Simulation/Evaluation, Modeling/Simulation/Evaluation, Modeling/Simulation/Evalua- 
n, Codes, Standards, Codes, Standards, Restructuring, Codes, Standards, Regulation, tion, Codes, Standards, 
Restructuring, Regulation, Regulation, Pricing) Pricing, Pollution Taxes, Green Regulation, Pricing, Land, Water, 
Pricing) Design) Energy, Biodiversity, Pesticides) 
¢ Public Information and ¢ Public Information and ¢ Public Information and ¢ Public Information and 
Education Education Education Education 
* New Sources (Atmospheric, © New Sources (Fossil, Solar, ¢ Rejuvenation (Ecosystem © New Sources (Agriculture, 
Surface, Ground) Wind, Hydropower, Geothermal, Remediation/Management) Aquaculture) 
Nuclear, Waste Product, 
Biomass) 
¢ Pollution Control (Storm ¢ Pollution Control (Hazardous ¢ Pollution Control (Air, Water, * Pollution Control (Pest, Disease, 
Water, Receiving Water, Waste Emission Tax, Renewable Land, Land Use, Biological, Solar, Deforestation, Desertification, 
© INCREASE Water, Industrial EfMluents, Sources, Energy Efficiency) Acid Rain, Desertification, ; Radiation) 
SUPPLY Cc hemical Pollutants, Greenhouse Gases, Deforestation, 
Agricultural Pollutants) Toxic Chemical, Radiation) 
« New Technologies (Water © New Technologies (Solar Cells, « New Technologies (Hazardous © New Technologies (Genetic 
Storage, Bioremediation, Fuel Cells, HVAC, Waste Transportation/Storage, Modification, Irrigation, 
Transportation, Hydraulics, lransportation, Drilling, Earthquakes, Volcanoes, Processing, Inspection, 
Water Treatment, Desalination) Refining, Mining, Cyclones, Hurricanes, Tornadoes, Preservation, Storage, 
Telecommuting, Production, Droughts, Fires, Floods, Distribution) 
Delivery, Cogeneration) Radiation, Diseases) 


Figure 1. Example program and policies to sustain life support systems 
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In reviewing the example life support programs and policies in Figure 1, several 
comments should be made. First, some programs and policies are focused on mitigating 
the ravages of nature. Interestingly, in certain cultures, it is assumed that anything 
natural must be safe or benign; unfortunately it should not be forgotten that humanity 
has spent most of its history attempting to protect itself from the destructive forces (e.g., 
earthquakes, hurricanes) of nature. Indeed, so-called organic food (i.e., grown in animal 
manure) can contain several virulent bacteria, including E. coli and salmonella. Second, 
it is obvious that only brief descriptions of programs and policies are contained in 
Figure 1. Moreover, each description may reflect a range of possibilities. As an example, 
“green design” could describe a system oriented, life cycle approach to producing, 
warehousing, selling, using, and, finally, recycling a product, or it could just be focused 
on the product’s biodegradable material. Third, the complexity of the programs and 
policies are also not reflected in Figure 1. As examples, the maintenance of economic 
growth by substituting technology for energy and other scarce resources and the 
development of “sinks” for greenhouse gases are both large-scale and complex in their 
undertaking. Such complexities are further aggravated by the fact that the four life 
support components interact in substantive ways. Water is needed to develop energy 
(e.g., acid mine drainage, hydropower), to sustain the environment (e.g., aquifers, 
climate cycles), and to grow food (e.g., agriculture, aquaculture); energy creation is, for 
the most part, detrimental to the environment (e.g., global warming, nuclear waste 
repositories); the environment sustains life by providing the raw material for water (e.g., 
oceans, reservoirs), energy (e.g., fossil fuels, solar energy), and food (e.g., wetlands, 
croplands); and food is the output of a judicious combination of water, energy and the 
environment’s bounty. 


Interestingly, from a global perspective, water—or the lack thereof—is becoming a very 
serious issue and is threatening the world’s food security. In short, water is being 
diverted from land irrigation to meet the needs of fast-growing urban centers and 
industries. (In regard to.consuming water, agriculture is simply more voracious than 
industry; in China, for example, a thousand tons of water produces one metric ton (ca. 
2000 pounds) of wheat, while the same amount of water used in industry yields an 
output worth many times the value of one metric ton of wheat.) China and other 
countries (e.g., in Africa, the Middle East) that are facing a water shortage may, of 
course, import food items; however, it is becoming more costly as the world’s grain 
reserves have decreased over the last decade. Instead, new programs—including 
technologies—and policies are being introduced in water management (e.g., 
conservation, pollution control, treatment, desalination) and food production (e.g., 
irrigation efficiency, yield through genetic modification, processing, preservation). It is 
obvious that the efficacy of such programs and policies must be carefully evaluated or 
analyzed, especially as the water shortage and food security problems become 
increasingly exacerbated as the world’s population grows by 50 percent in the next 50 
years (i.e., from 6 billion to 9 billion) and as the population prospers and demands a 
larger and better diet. 


Does the program or policy work? Is it worth the cost? Can and should it be 
implemented elsewhere? It is the reputed purpose of evaluation to provide answers to 
these and related questions. The need for conducting evaluations becomes more critical 
as programs/policies become more complex and more costly and, concomitantly, as the 
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tax base or resources for their funding remain fixed or decrease. Unfortunately, 
program/policy evaluation has not lived up to expectations. The field of evaluation is 
littered with efforts that do not adequately address the important issues or objectives, 
that do not employ valid controls for comparison purposes, that rely on inadequate 
measures or include expensive collections of data on measures that are in fact never 
used in the evaluation, that rely on inappropriate measurement methods, or that employ 
inadequate analytic techniques. Most, if not all, of the above-cited problems could be 
mitigated by developing, at the beginning of an evaluation effort, a valid and 
comprehensive evaluation design. 


Although there is no stock evaluation design that can be taken off the shelf and 
implemented without revision, there should be an approach or process by which such 
designs can be developed. Indeed, the author has identified a systems approach—that is 
at once purposeful, systematic, and consistent—for developing valid and comprehensive 
evaluation designs—it has since been successfully employed in a number of evaluation 
efforts. The approach is outlined in the next section, followed by a discussion of the 
elements that constitute an effective evaluation design. Several remarks are made at the 
conclusion of this paper. 


2. Evaluation Approach 


The proposed evaluation approach is defined below in terms of its process and its 
framework, followed by a discussion of threats to validity. 
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